feat: add voice transcript bridge and plugin dispatch infrastructure#1612
feat: add voice transcript bridge and plugin dispatch infrastructure#1612MomiJiSan wants to merge 4 commits into
Conversation
Add Neko core infrastructure for companion learning plugins: - Voice transcript event bridge (main_logic/agent_event_bus.py): routes real-time voice transcripts to plugins via custom events - Plugin dispatch service (plugin/server/.../dispatch_service.py): supports cancel_response / prime_context event contracts - Voice bridge action handling in agent_server.py - Voice bridge registration wiring in main_server.py - Hosted TSX check update for study_companion surfaces The boundary is intentionally narrow: the host only forwards events and executes generic session actions. All business logic — filtering rules, action definitions, learning scenarios — stays in the plugin layer (see companion PR for study_companion plugin). This is Part 1 of the original PR Project-N-E-K-O#1582 split.
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Repository UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (4)
🚧 Files skipped from review as they are similar to previous changes (4)
Walkthrough该 PR 实现语音转写桥接：从发布请求、agent 并发触发插件订阅并仲裁返回，到将结果回填主服并在 core 会话快照一致性下执行动作喵。 ChangesVoice Transcript Bridge System
其他
Sequence Diagram(s)sequenceDiagram
participant Core as Core
participant EventBus as EventBus
participant Agent as AgentServer
participant Dispatch as PluginDispatchService
participant Main as MainServer
Core->>EventBus: publish_voice_transcript_request_reliably(event_id, transcript, metadata)
EventBus->>Agent: voice_transcript_request(event_id, transcript, lanlan_name, metadata)
Agent->>Dispatch: trigger_custom_event_subscribers(event_type, args, timeout)
Dispatch-->>Agent: per-plugin dispatch results[]
Agent->>Main: emit voice_bridge_result(event_id, result)
Main->>EventBus: notify_voice_bridge_result(event_id, result)
EventBus-->>Core: resolve waiting Future -> return action/result
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 4✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0bb0d65f72
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| app.include_router(game_router) | ||
| app.include_router(card_assist_router) | ||
| app.include_router(capture_router) |
There was a problem hiding this comment.
When the main app is started, the four /api/card-assist/* endpoints defined in main_routers/card_assist_router.py are no longer included because the router import/include was removed from this router list. I checked the repo and these endpoints are still referenced by card-assist tests and the Character Card Manager feature, so users clicking the AI-assisted card authoring flow will now get 404s from the real app/main_server.py app rather than reaching the existing handlers.
Useful? React with 👍 / 👎.
| cm = get_config_manager() | ||
| ok, reasons = cm.is_agent_api_ready() | ||
| # 字段名保留 is_free_version（前端/下游 gate 消费者沿用），值取 agent 维度的 | ||
| # is_agent_free()：判 agent 是否走内置免费模型，而非 core/assist 的版本免费。 | ||
| return {"ready": ok, "reasons": reasons, "is_free_version": cm.is_agent_free()} | ||
| return {"ready": ok, "reasons": reasons, "is_free_version": cm.is_free_version()} |
There was a problem hiding this comment.
In configurations where the voice/core side is free but the Agent model has been switched to a paid/custom model, this now reports is_free_version: true even though the Agent path is not using the built-in free agent model. ConfigManager.is_agent_api_ready() documents that Agent free/quota UI state is supposed to come from is_agent_free(), and the main router still uses cfg.is_agent_free() for the same field, so this endpoint can re-enable the free-agent quota/warning behavior incorrectly for paid Agent setups.
Useful? React with 👍 / 👎.
| from plugin.plugins.study_companion.voice_contracts import ( | ||
| arbitrate_voice_transcript_results, | ||
| ) |
There was a problem hiding this comment.
When any voice_transcript subscriber returns a successful action, this import runs, but plugin/plugins/study_companion/voice_contracts.py is not present in the repo (searched for *voice_contracts.py and arbitrate_voice). The resulting ModuleNotFoundError is caught by _handle_voice_transcript_request as dispatch_failed, so valid plugin actions are discarded and the new voice bridge cannot actually arbitrate successful subscriber responses.
Useful? React with 👍 / 👎.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@app/agent_server.py`:
- Around line 1683-1720: The host code in _handle_voice_transcript_request
currently imports arbitrate_voice_transcript_results from
plugin.plugins.study_companion.voice_contracts, which couples the host to a
specific plugin; remove that import and either 1) move the arbitration logic
into the host/shared contract layer and call a locally defined
arbitrate_voice_transcript_results implementation, or 2) change the dispatch
layer so dispatch_results already returns a host-consumable standard result
(i.e., include normalized success/result/plugin_id/event_id fields) so no
plugin-specific import is needed; update _handle_voice_transcript_request to use
the new local/shared arbitration function or the normalized dispatch output and
ensure arbitration_items is computed the same way but without importing from
plugin.plugins.study_companion.voice_contracts.
In `@main_logic/agent_event_bus.py`:
- Around line 576-600: The timeout race happens because the recv thread enqueues
voice_bridge_result via run_coroutine_threadsafe but the main loop may not yet
call notify_voice_bridge_result(), so the waiter can be removed before resolve;
fix by adding an intermediate "queued" marker and checking it in the
timeout-retry logic: introduce a new set (e.g. _voice_bridge_waiters_queued) and
have the recv thread mark event_id as queued before calling
run_coroutine_threadsafe (and clear it when notify_voice_bridge_result() runs);
update the timeout branch around waiter/_voice_bridge_waiters_resolving to also
treat queued events as candidates for the short extra wait so the second wait
covers the period between enqueue and actual notify; ensure all places that
remove waiters (_voice_bridge_waiters) also clear queued/resolving safely under
_voice_bridge_waiters_lock.
In `@main_logic/core.py`:
- Around line 2015-2027: The code injects plugin result "context" into
prime_context without running apply_role_placeholders, causing placeholders like
{MASTER_NAME}/{LANLAN_NAME} to be left unexpanded for this pathway; modify the
block handling result.get("context") so that before calling
session_snapshot.prime_context(context_text, ...), you first call
apply_role_placeholders(context_text, session_snapshot or current session) and
use its returned text; keep checks for _session_changed(), respect skipped flag,
and only call prime_context if prime_context is callable and the
placeholder-applied context_text is non-empty.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 47550716-33ea-4c96-bba4-01b6f7ff267e
📒 Files selected for processing (10)
app/agent_server.pyapp/main_server.pyfrontend/plugin-manager/scripts/check-hosted-tsx.mjsmain_logic/agent_event_bus.pymain_logic/core.pyplugin/server/application/plugins/dispatch_service.pyplugin/tests/unit/server/test_plugin_dispatch_service.pytests/unit/test_agent_server_voice_bridge.pytests/unit/test_core_game_route_memory_contract.pytests/unit/test_voice_bridge_event_bus.py
| from plugin.plugins.study_companion.voice_contracts import ( | ||
| arbitrate_voice_transcript_results, | ||
| ) | ||
|
|
||
| arbitration_items: list[dict[str, object]] = [] | ||
| failure_count = 0 | ||
| for item in dispatch_results: | ||
| if not isinstance(item, Mapping): | ||
| continue | ||
| if not bool(item.get("success")): | ||
| failure_count += 1 | ||
| continue | ||
| result = item.get("result") | ||
| if not isinstance(result, Mapping): | ||
| continue | ||
| action = str(result.get("action") or "").strip() | ||
| if not action: | ||
| continue | ||
| payload: Dict[str, Any] = dict(result) | ||
| payload["action"] = action | ||
| plugin_id = str(item.get("plugin_id") or "").strip() | ||
| if plugin_id: | ||
| payload.setdefault("source_plugin", plugin_id) | ||
| source_event_id = str(item.get("event_id") or "").strip() | ||
| if source_event_id: | ||
| payload.setdefault("source_event_id", source_event_id) | ||
| arbitration_items.append( | ||
| { | ||
| "plugin_id": payload.get("source_plugin") or plugin_id, | ||
| "event_id": payload.get("source_event_id") or source_event_id, | ||
| "success": True, | ||
| "result": payload, | ||
| } | ||
| ) | ||
|
|
||
| if not arbitration_items: | ||
| return _voice_bridge_noop("no_handler_result", failures=failure_count) | ||
| payload = arbitrate_voice_transcript_results(arbitration_items) |
There was a problem hiding this comment.
把仲裁逻辑从具体插件包里拆出来喵。
这里在 host 侧直接依赖 plugin.plugins.study_companion.voice_contracts，会把这条 Part 1 的通用桥接链路运行时绑死到 Part 2 的 study_companion 实现上喵。只要有任意 voice subscriber 返回结果、但这个插件模块没有随当前部署一起存在，_handle_voice_transcript_request() 就会在这里抛错并回落成 dispatch_failed，整个 voice bridge 就没法独立工作了喵；当前这个 PR 里的 happy-path 单测也会被同一个依赖一起绑住喵。这个仲裁逻辑应该放到 host/shared contract 层，或者让 dispatch 层直接返回 host 可消费的标准结果，而不是从具体插件包反向 import 喵。
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@app/agent_server.py` around lines 1683 - 1720, The host code in
_handle_voice_transcript_request currently imports
arbitrate_voice_transcript_results from
plugin.plugins.study_companion.voice_contracts, which couples the host to a
specific plugin; remove that import and either 1) move the arbitration logic
into the host/shared contract layer and call a locally defined
arbitrate_voice_transcript_results implementation, or 2) change the dispatch
layer so dispatch_results already returns a host-consumable standard result
(i.e., include normalized success/result/plugin_id/event_id fields) so no
plugin-specific import is needed; update _handle_voice_transcript_request to use
the new local/shared arbitration function or the normalized dispatch output and
ensure arbitration_items is computed the same way but without importing from
plugin.plugins.study_companion.voice_contracts.
fix: resolve plugin boundary violation and improve voice bridge robustness - CRITICAL: Move arbitrate_voice_transcript_results from plugin/plugins/study_companion/voice_contracts.py to plugin/server/application/plugins/voice_contracts.py (shared infrastructure). Update import in agent_server.py accordingly. This fixes the plugin boundary violation where app/ imported from a specific plugin (备忘录 §2). - Add explicit asyncio.CancelledError re-raise before broadcast Exception handlers in agent_server.py, core.py, and dispatch_service.py (matches codebase conventions). - Add asyncio.TimeoutError-specific handling in dispatch_service.py to distinguish handler timeouts from crashes. - Extract _validate_timeout() helper in dispatch_service.py to eliminate duplicated timeout validation between trigger_custom_event and trigger_custom_event_subscribers. - Remove redundant _session_changed() calls in _dispatch_voice_transcript_bridge (core.py) — the outer check at the top already guards both cancel_response and prime_context branches. Inner checks after await remain as necessary. @
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a9d3b6ec8a
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| self._activity_tracker.on_voice_rms() | ||
|
|
||
| if is_voice_source and transcript_text: | ||
| voice_bridge_action = await self._dispatch_voice_transcript_bridge(transcript_text) |
There was a problem hiding this comment.
When the main app has started its local PUB socket but agent_server is not connected/running, publish_session_event() still returns true because it only sends to the local ZeroMQ socket and cannot know there are no subscribers; this awaited call then waits the full publish_voice_transcript_request_reliably timeout (default 1.2s) before ordinary transcript handling continues. In that environment every realtime voice transcript is delayed by the bridge even though the plugin result can never arrive, so the fallback path should not synchronously block the voice hot path without a real agent-side readiness/ack signal.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
🧹 Nitpick comments (2)
tests/unit/test_agent_server_voice_bridge.py (1)
227-257: ⚡ Quick win这个用例名写了“before_dispatch”，但断言还没有验证先后顺序喵。
现在只检查
voice_bridge_request_seen最终出现；就算_handle_voice_transcript_request先跑，这个测试也会通过，回归可能被漏掉喵。建议显式记录执行顺序并断言seen早于handle喵。🔧 可参考的最小改法
@@ - emitted: list[tuple[str, str | None, dict[str, Any]]] = [] + emitted: list[tuple[str, str | None, dict[str, Any]]] = [] + order: list[str] = [] @@ async def _emit_main_event(event_type: str, lanlan_name: str | None, **payload: Any) -> None: emitted.append((event_type, lanlan_name, payload)) + order.append(f"emit:{event_type}") @@ async def _handle_voice_transcript_request(_event: dict[str, Any]) -> None: + order.append("handle") handled.set() @@ assert ("voice_bridge_request_seen", "Yui", {"event_id": "voice-seen"}) in emitted + assert order.index("emit:voice_bridge_request_seen") < order.index("handle")🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/unit/test_agent_server_voice_bridge.py` around lines 227 - 257, The test name implies verifying ordering but currently only checks presence; modify the monkeypatched helpers to record execution order and assert that "seen" occurs before "handle": update the _emit_main_event spy (patched onto srv._emit_main_event) to append a tuple including a marker like ("seen", event_type, lanlan_name, payload) into the emitted list when event_type == "voice_bridge_request_seen", and update the _handle_voice_transcript_request spy (patched onto srv._handle_voice_transcript_request) to append a marker like ("handled", ...) before setting the handled event; after calling srv._on_session_event, assert that the index of the "seen" entry in emitted is less than the index of the "handled" entry to ensure seen was emitted before handling.tests/unit/test_voice_bridge_event_bus.py (1)
116-117: ⚡ Quick win这里的清理断言建议覆盖全部 voice-bridge waiter 状态喵。
现在只断言
_voice_bridge_waiters == {}，如果_voice_bridge_request_seen_waiters、_voice_bridge_waiters_queued或_voice_bridge_waiters_resolving泄漏，会漏检喵。建议抽一个统一 helper，在这些发布成功/超时用例末尾复用喵。🔧 可参考的断言 helper
+def _assert_voice_bridge_waiters_cleared() -> None: + assert agent_event_bus._voice_bridge_waiters == {} + assert agent_event_bus._voice_bridge_request_seen_waiters == {} + assert agent_event_bus._voice_bridge_waiters_queued == set() + assert agent_event_bus._voice_bridge_waiters_resolving == set() @@ - assert agent_event_bus._voice_bridge_waiters == {} + _assert_voice_bridge_waiters_cleared()🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/unit/test_voice_bridge_event_bus.py` around lines 116 - 117, 当前测试仅断言 agent_event_bus._voice_bridge_waiters == {}，会漏掉其他 waiter 状态泄露；请在 tests/unit/test_voice_bridge_event_bus.py 抽出一个统一 helper（例如 assert_no_voice_bridge_waiters(agent_event_bus)）来同时检查 _voice_bridge_waiters、_voice_bridge_request_seen_waiters、_voice_bridge_waiters_queued 和 _voice_bridge_waiters_resolving 都为空或处于初始状态，然后在所有发布成功/超时用例末尾调用该 helper 来复用断言并覆盖全部 voice-bridge waiter 状态。
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@tests/unit/test_agent_server_voice_bridge.py`:
- Around line 227-257: The test name implies verifying ordering but currently
only checks presence; modify the monkeypatched helpers to record execution order
and assert that "seen" occurs before "handle": update the _emit_main_event spy
(patched onto srv._emit_main_event) to append a tuple including a marker like
("seen", event_type, lanlan_name, payload) into the emitted list when event_type
== "voice_bridge_request_seen", and update the _handle_voice_transcript_request
spy (patched onto srv._handle_voice_transcript_request) to append a marker like
("handled", ...) before setting the handled event; after calling
srv._on_session_event, assert that the index of the "seen" entry in emitted is
less than the index of the "handled" entry to ensure seen was emitted before
handling.
In `@tests/unit/test_voice_bridge_event_bus.py`:
- Around line 116-117: 当前测试仅断言 agent_event_bus._voice_bridge_waiters == {}，会漏掉其他
waiter 状态泄露；请在 tests/unit/test_voice_bridge_event_bus.py 抽出一个统一 helper（例如
assert_no_voice_bridge_waiters(agent_event_bus)）来同时检查
_voice_bridge_waiters、_voice_bridge_request_seen_waiters、_voice_bridge_waiters_queued
和 _voice_bridge_waiters_resolving 都为空或处于初始状态，然后在所有发布成功/超时用例末尾调用该 helper
来复用断言并覆盖全部 voice-bridge waiter 状态。
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI
Review profile: CHILL
Plan: Pro Plus
Run ID: c68d0250-44b0-4f38-a429-af107e2ac510
📒 Files selected for processing (8)
app/agent_server.pyapp/main_server.pymain_logic/agent_event_bus.pymain_logic/core.pytests/test_agent_rewrite_regression.pytests/unit/test_agent_server_voice_bridge.pytests/unit/test_core_game_route_memory_contract.pytests/unit/test_voice_bridge_event_bus.py
🚧 Files skipped from review as they are similar to previous changes (3)
- main_logic/core.py
- app/agent_server.py
- main_logic/agent_event_bus.py
这个 PR 做了什么这个 PR 给 Neko 本体新增了通用的 语音转录桥接（voice transcript bridge） 和 插件调度基础设施，11 个文件，不涉及任何具体插件内容。 核心流程：
为什么要这样做核心目标是把"伴学语音判断"的通道能力收敛到 Neko 本体，同时把业务决策留给插件。 原来的设计是宿主直接承载伴学业务语义（判断自言自语、要不要打断、注入什么上下文），这违反插件边界。正确边界应该像现在这样：
这样后续其他插件（不只是伴学）也可以注册
|
1 participant
概述 (Part 1/2 — Neko 本体)
变更范围（11 文件）
设计边界
host 只做通道：转发 voice transcript、等待插件结果、执行通用 session 操作。所有伴学业务逻辑在 Part 2 插件内。
审查修复记录
arbitrate_voice_transcript_results从plugin/plugins/study_companion/提升到plugin/server/application/plugins/voice_contracts.pyasyncio.CancelledErrorre-raiseasyncio.TimeoutError专项处理 +_validate_timeouthelper_session_changed()调用Summary by CodeRabbit
新功能
优化
测试